Adaptive Hierarchy-Branch Fusion for Online Knowledge Distillation

Online Knowledge Distillation (OKD) is designed to alleviate the dilemma that high-capacity pre-trained teacher model not available. However, existing methods mostly focus on improving ensemble prediction accuracy from multiple students (a.k.a. branches), which often overlook homogenization problem makes student saturate quickly and hurts performance. We assume intrinsic bottleneck of comes identical branch architecture coarse strategy. propose a novel Adaptive Hierarchy-Branch Fusion framework for Distillation, termed AHBF-OKD, designs hierarchical branches adaptive hierarchy-branch fusion module boost diversity aggregate complementary knowledge. Specifically, we first introduce architectures construct diverse peers by increasing depth monotonously basis target branch. To effectively transfer knowledge most complex simplest branch, an create assistants recursively, regards as smallest assistant. During training, assistant previous hierarchy explicitly distilled current hierarchy. Thus, important scores different are adaptively allocated reduce homogenization. Extensive experiments demonstrate effectiveness AHBF-OKD datasets, including CIFAR-10/100 ImageNet 2012. For example, 2012, ResNet-18 achieves Top-1 error 29.28\%, significantly outperforms state-of-the-art methods. The source code available at https://github.com/linruigong965/AHBF.